User-Oriented MT Evaluation and Text Typology

A brief survey of user-oriented MT evaluation methodologies suggests that they all suffer drawbacks in terms of time and/or generality and/or user interpretability. The time and cost of evaluation, coupled with system prices, make it increasingly less likely that full-scale pre-purchase evaluation by a single potential user will be economic. It is clearly desirable that MT systems should be “generically benchmarked” in some way to agreed standards. Regardless of the particular evaluation metrics used, generic benchmarking requires corpora of typed texts as test material, presupposing a shared notion of text type. Document structure, which is increasingly subject to international standardisation, provides a useful basis for text typing in this context. Furthermore, elements of a given document structure can be associated with linguistic diagnostics usable in statistical measures of prototypicality. 1 User-Oriented Evaluation Potential users of an MT system want to know whether it is likely to be an economical factor in the translation of their texts. Test methodologies that we know about use either prototypical samples of end-user texts or artificial texts constructed by designers and developers, such as a suite of test sentences designed to exhibit linguistic constructions on a “one-per-sent” basis (Flickinger, 1987). From an end-user perspective, artificial texts are not necessarily the ideal basis for system evaluation: • For a linguistically naive user, specifying that System X translates gapping constructions successfully is not informative. Even if s/he is shown an example of a gapping construction, it is not immediately obvious what strings are to count as gapping constructions.